I've ran a few experiments after on-demand syncing >10,000 package from centos and here are some conclusions:

1. The distinct query suggested by @dralley is better! [0]
2. The extract_pk performs very poorly with pulp_hrefs because of the use of django's resolve. [1]

[0] Query comparision w/ list of 10,000 packages prns

In [54]: timeit(lambda: Content.objects.filter(pk__in=(extract_pk(n) for n in prns)).values("pulp_domain").distinct(), number=10)
Out[55]: 0.17054506699787453

In [56]: timeit(lambda: Content.objects.filter(pk__in=(extract_pk(n) for n in prns),pulp_domain=domain).count(), number=10)
Out[57]: 0.6148349700015387

[1] Very bad performance with extract_pk from pulp_href

In [34]: timeit(lambda: [extract_pk(n) for n in prns], number=10)
Out[34]: 0.023607832001289353

In [35]: timeit(lambda: [extract_pk(n) for n in hrefs], number=10)
Out[35]: 22.53728126399801

In [36]: len(prns)
Out[36]: 10000

In [37]: len(hrefs)
Out[37]: 10000

Great!

If we can do all validation in the view layer with clever queries, that would be greatly preferable to deferring it to the task.

If we can reimplement extract_pk such that it doesn't use Django's resolve(), but still do the validation we want to do, that would be ideal.